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Overview 



The HOPE Foundation (HOPE) commissioned the American Institutes for Research (AIR) to 
conduct an exploratory study of the implementation and impact of the Courageous Eeadership 
Academy (CEA). In this report, we introduce the school reform model, describe the study 
methodology, present findings for each of the three research questions explored, and discuss the 
implications for the intervention and additional research. 

This school reform model builds on the principles in Failure is Not an Option (FNO) Six 
Principles to structure and inform the thoughtful development and use of School Eeadership 
Teams (SET) functioning as professional learning communities (PEC) and embedded 
professional development to build school staff as change leaders. The FNO Six Principles' are: 

1. Developing core mission, vision, values, and goals 

2. Ensuring achievement for all students: Systems for intervention and prevention 

3. Collaborative teaming focused on teaching and learning 

4. Using data to guide decision making and continuous improvement 

5. Gaining active engagement from family and community 

6. Building sustainable leadership capacity 

CEA involves a three-phase process of (1) taking stock, learning the FNO Six Principles, and 
building a professional learning community; (2) developing and implementing a plan to improve 
instruction; and (3) institutionalizing the practices (see Exhibit 1). Full implementation is 
designed to occur over three years, and involves selected staff from participating schools 
working across the district and within their home schools. 

The first step is to develop an accurate understanding of the current state of readiness, needs, and 
goals in the district as well as the commitment of district and site leadership to provide resources 
to support full implementation. After setting goals in the first CEA session, the leadership team 
revisits Instructional SMART Goal checkpoints, held during each of the four CEA sessions per 
year. Each school leadership team, facilitated by HOPE faculty, identifies high priority 
Instructional SMART Goals, and discusses data and progress towards their goals. A critical 
agenda item at this meeting is to monitor progress of the CEA participants and make customized 
adjustments, as indicated by their data. 

In the first year, “CEA I,” school leadership teams from participating schools join into a district- 
wide professional learning community to learn together about the FNO Six Principles and work 
together implementing them. The four professional learning sessions that year cover: (1) setting 
the stage for accelerated learning; (2) focusing on collaborative teaming and instruction; (3) 
building a professional development (PD) plan and communicating progress; and (4) reflecting 
on what participants have learned and reporting results. 

In the second year, “CEA II,” school leadership teams integrate additional tools and strategies in 
planning at their schools. The school leadership teams usually consists of 5-8 members. 



* See Appenix 1 for the Hope Foundation’s description of these principles. 

^ SMART goals are “specific, measurable, attainable, realistic, and timely.” 
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including the principal and department leads (in middle and high schools), and other teachers 
identified as leaders by the district and site administrators along with HOPE faculty. The four 
professional learning sessions that year cover: (1) becoming a high performing team; (2) 
engaging in collegial practices for instructional improvements; (3) communicating progress 
monitoring; and (4) maintaining FNO Six Principles to sustain high performing teams. 

In the third year, “CLA III,” the school leadership teams work on institutionalizing the learning 
and structures developed the previous year. The four professional learning sessions that year 
cover: (1) leading the school’s PLC; (2) facilitating change to achieve commitment; (3) 
sustaining leadership across the school and district; and (4) sustaining the collegial culture. Each 
professional learning session lasts a full school day («8am-3pm), and is led by two HOPE faculty 
members: a faculty lead and a project lead. The faculty lead is responsible for facilitating the 
CEA session, while the project lead coordinates the planning and facilitation of the session in 
addition to working with the district and site administrators to provide ongoing support. 

The HOPE Eoundation faculty work closely with leadership teams from multiple schools within 
the district, and supports the leadership teams in developing staff in their home schools. 

The anticipated ultimate outcome of CEA is improved student achievement. CEA is designed to 
improve student achievement by creating a collaborative professional community that functions 
to improve instruction and school climate. Therefore, changes in teacher collaboration, 
instruction, and other conditions for learning could be considered proximal outcomes. 



Exhibit 1: Courageous Leadership Academy Theory of Action 




^ For more information on the Corageous Leadership Academies, see 
http://www.hopefoundation.org/programs/courageous-leadership-academv.html 
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Methodology 

This exploratory study is designed to address the following research questions: 

• Research Question 1: To what extent have school staff built a professional learning 
community? 

• Research Question 2: Does higher level of implementation correspond to higher 
achievement? 

• Research Question 3: Has student achievement improved? 

The study examines extant data to identify potential relationships among CLA implementation 
and student scores. As such, it is limited to retrospective data available from the sites 
implementing CLA. The study sample contains 34 schools in the Mansfield Independent School 
District, in Texas. Almost all schools in this district participate in the CLA, although they began 
at different times. For this study, we consider two cohorts of schools: the 14 intermediate, 
middle, and high schools that began implementing the CLA in 2007-08 (Cohort 1) and the 20 
elementary schools that began implementing in 2008-09 (Cohort 2).^ 

This study examines two types of extant data: the Failure is Not an Option (FNO) Readiness 
Survey and the school-level percent proficient on the Texas Assessment of Knowledge and Skills 
(TAKS) mathematics and reading scores. To look at progress towards building a professional 
learning community, we drew on data from the Failure is Not an Option (FNO) Readiness 
Survey, administered by the HOPE Foundation as part of the intervention. The FNO Readiness 
Survey asks four questions for each of the FNO Six Principles. Two questions relate to 
understanding and use of the principle (Knowledge/Skill Development) and two relate to 
readiness to implement protocols, new practices, and strategies to advance priority goals 
(Commitment and Action).^ 

In the 2009-10 school year, HOPE staff began administering this survey to school leadership 
team members in CLA schools twice annually, once in Fall 2009 (“pre-test” survey) and once in 
Spring 2010 (“post-test” survey). Because all the schools in the sample have been implementing 
the CLA for at least a year before the FNO was administered, we could not consider either 
survey a “pre-test”, since neither were administered before implementation. Therefore, we 
focused only on the “post-test” survey administered in Spring 2010. 

As with any study using extant data, interpretations are limited by the nature of the data. For 
example, a pre-intervention measure of the FNO Readiness Survey was not available, so it is not 
possible to look at changes in understanding and use of CLA over time, or to make conclusive 
statements about causality. Because the study was limited to existing measures, some aspects of 
implementation and types of outcomes (e.g., schoolwide changes in conditions for learning) 
could not be analyzed in this exploratory study. 



While there are currently 40 schools in the Mansfield Independent School District, only the 34 schools with FNO 
survey data and with student data from 2008-09 and 2009-10 were included. 

Appendix 4 contains demographic information on Mansfield ISD, and Texas. 

^ Note that some schools which opened after CLA began started using CLA somewhat later than the rest of their 
cohort. See Appendix 2 for a brief summary of the sample. 

® Information on scale reliability was not available. 
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For student achievement data, Mansfield ISD provided AIR with student level data sets for six 
years: 2004-05, 2005-06, 2006-07, 2007-08, 2008-09 and 2009-10. The student data included 
the students’ school name, grade, and the Texas Assessment of Knowledge and Skills (TAKS) 
mathematics and reading scores. In Texas, students are tested in grades 3-12. Therefore, we had 
test score data for grades 3-4 in elementary schools, 5-6 for intermediate schools, 7-8 for middle 
schools, and 9-12 for high schools. 

These scores were provided in three formats: raw score, scale score, and a “met standard” 
indicator variable. Many studies of school reforms use scale scores as their outcomes. In this 
case, changes in the scaling in 2009-10 complicate longitudinal analysis of scale scores. The 
percent proficient metric, which is calculated determining the percent of students in the school 
who “met standards” is acceptable for longitudinal analysis.^ Therefore, this analysis used two 
percentage based proficiency measures: percent of the school proficient in reading and percent 
proficient in mathematics. If a student has met the state standard in mathematics or reading, then 
the state assumes “student has learned, understood, and is able to apply the important concepts 
and skills expected at each tested grade level. 
(http://ritter.tea.state.tx.us/perfreport/src/2010/definitions.html) ” 



Research Question 1: To what extent have school staff built a professional learning community, 
including: 



a. learning and internalizing the FNO Six Principles of the CLA? 

b. achieving common purposes, goals, and values? 

c. building a culture of trust, mutual accountability, and continuous 
improvement? 

First, this study looks at progress towards building a professional learning community, using data 
from the FNO Readiness Survey. Scores on this survey indicate readiness in terms of (1) 
Knowledge and Skills Development and (2) Commitment and Action. 

Table 1.1 presents the descriptions of the Knowledge and Skills, and the Commitment and 
Action Readiness Levels. Both scales have a range of 0 to 48. Tables 1.2 and 1.3 present the 

o 

total number of schools in Mansfield ISD, by cohort, by readiness level. 



^ Percent proficient is not affected by the change in the scale schools. For example, if the original scale range was 200- 
1500 (range) and had a “proficient” cutpoint of 1100, and the rescaled scale range was200-800 with a cutpoint of 650, a 
student who achieved just above the cutpoint (was “proficient”) with a score of 1100 on the first scale would have 
earned a score of 650 (was “proficient”) on the second scale. 

^ For a full list of each schools readiness scores, as well as their Knowledge and Skills, and their Commitment and 
Action readiness levels, see Appendix 3. 
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Table 1.1 Descriptions of Knowledge/Skills and Commitment/Action Readiness Levels 



Knowledge/Skills 

Level & Scores 


Ability Level 

Descriptions 


Frustration Level 
12 or less 


Learners lack experience and an adequate understanding 
of the background knowledge and skills to perform or 
engage at a meaningful level 


Awareness & Learning 
13 to 24 


The learner has some initial understanding of the 
foundational concept/knowledge 


Partial Work Environment 

Integration 

25 to 36 


The learner has had some experience with the concept 
and exhibits application in their work. 


Infused into the Work Culture 
37 to 48 


The learner is at a high level of integration and has 
incorporated a concept and skill into their repertoire. 


Commitment/Action 

Level & Scores 


Willingness Level 

Descriptions 


Direction to begin process 
12 or less 


The leader needs explicit direction to understand and 
apply the concept. Their current ability level can develop 
further with direct instruction and short-term goals - 
followed by observation and feedback. 


Coaching and mediation to 
practice and learn 
13 to 24 


The leader is about to take some initiative for facilitating 
and engaging others, supported by dialogue and feedback 
from a coaching relationship. This could be part of a 
team effort or one-on-one coaching depending, on the 
circumstances. 


Support and follow-up to 
sustain progress 
25 to 36 


The leader has successfully implemented new practices 
and exhibited a high level of commitment to the shared 
purpose and mission. 


Check-points to maintain and 
continually improve 
37 to 48 


The leader has advanced to a level of independence. 



Table 1.2: Total number of schools for Knowledge and Skills, by Cohort 





Cohort 1 


Cohort 2 


Frustration Level 


0 


0 


Awareness & Learning 


0 


0 


Partial Work Environment Integration 


5 


2 


Infused into the Work Culture 


9 


18 
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Table 1.3: Total number of schools for Commitment and Action, by Cohort 





Cohort 1 


Cohort 2 


Direetion to begin proeess 


0 


0 


Coaehing and mediation to practice and learn 


0 


0 


Support and follow-up to sustain progress 


4 


1 


Check-points to maintain and continually improve 


10 


19 



Findings 

By Spring 2010, most CLA schools in Mansfield indicated very high levels of knowledge and 
skills, as well as commitment and action, for the core principles. No school scored below the 
third (of fourth) levels on either Knowledge and Skills Readiness Levels or Commitment and 
Aetion Readiness Levels. 

• 64% (9/14 sehools) of Cohort 1 seored at the top level for knowledge and skills 

• 71% (10/14 schools) of Cohort 1 scored at the top level for commitment and action 

• 90% (18/20 schools) of Cohort 2 scored at the top level for knowledge and skills 

• 95% (19/20 schools) of Cohort 2 scored at the top level for commitment and action 

Only four of 34 schools — Emma Nash, Danny Jones, Mansfield, and Mansfield Timberview — 
seored below the top level for both measures 

Looking aeross groups of sehools, we find that Cohort 2 sehools appeared to be more eonsistent 
about implementing at high levels. This may appear eounterintuitive, given that Cohort 1 had an 
additional year to reach high levels of implementation. We suggest — ^but cannot test with this 
data set — several possible explanations: 

• Cohorts are eonfounded with sehool level. The higher level of implementation for Cohort 
2 might reflect differences in implementation in elementary school (Cohort 2) versus 
intermediate, middle, and high school (Cohort 1). Educational research suggests that 
implementing reforms is more diffieult in middle and high schools than in elementary 
sehools (U.S. Department of Education, 2005). To test this hypothesis, additional analysis 
might look at CEA implementation in districts where cohorts are not defined by school 
levels. If this is the issue, HOPE might consider additional measures to support 
implementation in upper sehool levels. 

• Early implementers might lose energy after several years of implementation. CEAs 
require substantial eommitment and work on the part of the school leadership teams and 
other staff. It may be that after several years, sehools lag in their effort. To test this 
hypothesis, additional analysis might look at the Mansfield schools in 2010-11. If Cohort 
2 shows a drop in implementation, this would reinforce the hypothesis. If this is the issue, 
HOPE might consider strategies to maintain and build enthusiasm at this critieal juneture. 

• A related hypothesis is that the district — and perhaps the HOPE Eoundation — improved 
proeesses and tools for implementation between the Cohort 1 and Cohort 2. Thus, the 
seeond eohort would benefit from the learning experiences of the first. The HOPE 
Eoundation may have information about the evolution of their tools and processes which 
lend credence to this hypothesis. If this is the issue, then the patterns of implementation 
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would suggest that changes to processes and tools were, in fact, improvements, and 
would validate HOPE’S current direction. 

Research Question 2: Does higher level of implementation correspond to higher achievement? 

School reform research typically shows that level of implementation moderates effects. 
Therefore, this study examines whether there is a relationship between the level of 
implementation and student outcomes. Using the data described in Research Question 1 and 
percent proficient metrics in mathematics and reading^, we considered the relationship between 
level of implementation and student achievement. Table 2.1 shows the correlations between 
mathematics test percent proficient and the two measures of implementation from the FNO 
Readiness Survey (Knowledge and Skills and Commitment and Action), for the period between 
school year 2004-05 and school year 2009-10. Table 2.2 shows the corresponding data for 
reading test percent proficient. In both tables, the period of CLA implementation is highlighted. 
That is the period in which one would expect the implementation measure to most strongly 
correlate to achievement. Finally, we examined the relationship between specific questions on 
the FNO Readiness Survey and achievement, to explore the relationships among those items 
most closely related to teacher community and student achievement. 



Table 2.1. Correlations between Percent Proficient in Mathematics with Knowledge and 
Skills Score and Commitment and Action Score, by Cohort 





2004-05 


2005-06 


2006-07 


2007-08 


2008-09 


2009-10 


Cohort 1 














Knowledge and Skills 








CLA I 


CLA II 


CLA III 


Coefficient estimate 


0.60 


0.69 


0.52 


0.51 


0.57 


0.25 


P Value 


0.05 


0.02* 


0.07 


0.06 


0.03* 


0.39 


Commitment and 
Action 














Coefficient estimate 


0.62 


0.69 


0.58 


0.52 


0.60 


0.30 


P Value 


0.04* 


0.02* 


0.04* 


0.06 


0.02* 


0.30 


Cohort 2 














Knowledge and Skills 










CLA I 


CLA II 


Coefficient estimate 


0.41 


0.10 


0.07 


0.26 


0.54 


0.41 


P Value 


0.15 


0.71 


0.80 


0.29 


0.01* 


0.07 


Commitment and 
Action 














Coefficient estimate 


0.39 


0.05 


0.03 


0.22 


0.48 


0.35 


P Value 


0.17 


0.87 


0.91 


0.37 


0.03* 


0.12 



Statistical significance at the p < .05 level is indicated by an asterisk (*). 



^ The two percent proficient metrics are described in more detail in the discussion on Research Question 3. 
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Table 2.2. Correlations between Percent Proficient in Reading with Knowledge and Skills 
Score and Commitment and Action Score, by Cohort 





2004-05 


2005-06 


2006-07 


2007-08 


2008-09 


2009-10 


Cohort 1 














Knowledge and Skills 








CLA I 


CLA II 


CLA III 


Coefficient estimate 


0.69 


0.24 


0.38 


0.26 


0.16 


-0.44 


P Value 


0.02* 


0.48 


0.21 


0.37 


0.58 


0.12 


Commitment and 
Action 














Coefficient estimate 


0.67 


0.19 


0.39 


0.25 


0.17 


-0.46 


P Value 


0.02* 


0.57 


0.19 


0.40 


0.55 


0.10 


Cohort 2 














Knowledge and Skills 










CLA I 


CLA II 


Coefficient estimate 


0.43 


0.31 


0.31 


0.27 


0.57 


0.48 


P Value 


0.13 


0.25 


0.21 


0.26 


0.01* 


0.03* 


Commitment and 
Action 














Coefficient estimate 


0.45 


0.26 


0.28 


0.25 


0.50 


0.42 


P Value 


0.10 


0.34 


0.26 


0.30 


0.03* 


0.07 



Statistical significance at the p < .05 level is indieated by an asterisk (*). 



We took a closer look at the relationship between Failure is Not an Option (FNO) Readiness 
Survey questions that asked specifically about professional learning communities (questions 5, 
10 and 12), which is the heart of the CLA model, and Cohort 2 schools, which appear to be the 
stronger implementers. 

The survey asks respondents how much do they agree or disagree with the following 
statements (strongly agree, agree, disagree, strongly disagree): 

Question 5: Teachers and administrators are competent in working with one another to 
continuously build and sustain Professional Learning Communities, focused on 
instructional improvement. 

Question 10: Teachers as learners consciously practice collaborative problem solving to 
improve professional learning outcomes. 

Question 12: Effective team strategies are used for continual team learning and 
improvement which are monitored over time to assess progress and inform next steps. 
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Table 2.3. Cohort 2 Correlations between Percent Proficient in Mathematics with FNO 
Survey Questions 5, 10 and 12 

2004-05 2005-06 2006-07 2007-08 2008-09 2009-10 



Question 5 



Coejficient estimate 


0.38 


-0.03 


P Value 


0.19 


0.92 


Question 10 


Coejficient estimate 


0.43 


0.26 


P Value 


0.13 


0.32 


Question 12 


Coefficient estimate 


0.45 


0.11 







CLA I 


CLA II 


0.04 


0.17 


0.38 


0.34 


0.88 


0.49 


0.09 


0.14 


0.31 


0.37 


0.59 


0.43 


0.21 


0.12 


0.01* 


0.06 


0.16 


0.42 


0.52 


0.51 


0.52 


0.07 


0.02* 


0.02* 



P Value 



0.11 



0.70 



Statistical significance at the p < .05 level is indieated by an asterisk (*). 



Table 2.4. Cohort 2 Correlations between Percent Proficient in Reading with FNO Survey 
Questions 5, 10 and 12 

2004-05 2005-06 2006-07 2007-08 2008-09 2009-10 

Question 5 CLA 1 CLA II 



Coefficient estimate 


0.39 


0.37 


0.23 


0.25 


0.39 


0.35 


P Value 


0.17 


0.16 


0.36 


0.31 


0.09 


0.13 


Question 10 














Coefficient estimate 


0.43 


0.36 


0.49 


0.32 


0.63 


0.57 


P Value 


0.12 


0.17 


0.04 


0.18 


0.00* 


0.01* 


Question 12 














Coefficient estimate 


0.42 


0.27 


0.31 


0.41 


0.66 


0.47 


P Value 


0.13 


0.32 


0.20 


0.08 


0.00* 


0.04* 


Statistieal signifieance at the p < .05 level is indicated by an asterisk (*). 



Findings 

Cohort 2 schools (which on average had higher levels of implementation than Cohort 1) that had 
agreed more with these questions also had higher levels of proficiency in mathematics and 
reading (See tables 2.3 and 2.4). These data are correlation and cannot establish a causal 
relation. However, they suggest that there may be a testable relationship between the 
development of professional learning communities and student achievement, which is consistent 
with some other research (Bryk & Driscoll, 1988; Lee & Smith, 1995, 1996). A study that looks 
prospectively at this relationship, with a rigorously matched comparison group, could provide a 
clearer picture of whether CLA professional learning communities, done well, improve student 
achievement, and if combined with meditational analyses and/or qualitative data may help 
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unpack the reason for such a relationship. 

The relationship between implementation and achievement was generally strong. Since the 
sample size was small, we did not expect to see statistically significant effects, even with very 
high correlations. During the years of implementation, correlations between implementation and 
mathematics achievement ranged from 0.25 to 0.60; during these same years, correlations 
between implementation and reading achievement ranged from -0.46 to 0.57.^° 

• For Cohort 1, implementation (measured in Spring 2010) had consistent, strong 
correlations with mathematics achievement (measured each year from 2005-2010). 

• For Cohort 2, implementation correlated with mathematics achievement better after CLA 
implementation than before. 

• For Cohort 1, implementation correlated with reading achievement better before CLA 
implementation than after. 

• For Cohort 2, implementation consistently correlated strongly with reading achievement. 

• For Cohort 2, schools that have agreed more with questions related to professional 
communities also had higher levels of proficiency in mathematics and reading. 

Correlations of extant data are particularly susceptible to misinterpretation. In cases where the 
implementation (measured in Spring 2010) correlated strongly with achievement over the six- 
year span, the direction of causality is especially unclear. It is possible that schools with high 
achievement are more open to implementing CLA than schools with low achievement. To truly 
test causality, it is important to conduct a prospective study using random or well matched 
comparison groups, and to consider pre- and post-measures of both the implementation and 
outcome variables. The case where the relationship between implementation and achievement is 
strongest in the implementation period (Cohort 2, mathematics) suggests the possibility that 
strong implementation relates to higher achievement. Again, a strong prospective study would 
provide more confidence in the findings. 



Research Question 3: Has student achievement improved? 

Ultimately, the goal of CLA is to improve student learning and achievement through improved 
professional community and teaching. To explore this question, the study examined changes in 
test scores in schools implementing CLA. 

Exhibit 3.1 presents mathematics achievement data, measured as percent proficient across the 
full sample of CLA schools, for Cohorts 1 and 2 from the 2004-5 school year through the 2009- 
10 school year. Exhibit 3.2 presents corresponding data for reading achievement. The state 
average is presented for comparison. 

Of special interest is the difference between achievement before CEA (up to Eall 2007 for 
Cohort 1 and Eall 2008 for Cohort 2) and achievement after the CEA had been fully 
implemented (Spring 2010 for Cohort 1 and Spring 2011 for Cohort 2). Both exhibits have 

*** The only negative correlations with reading achievement occurred during the 2009-10 school year with the Cohort 
1 schools, and were not significant. 
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indicators for when each cohort began CLA implementation. Due to the timing of this study, we 
can only examine outcomes for Cohort 2 mid-implementation (before the schools had gone 
through the full three years of implementation). 

Exhibit 3.1. Math Proficiency by Cohort from 2004-05 to 2009-10 




Cohort 1 Cohort 2 • State Avg. Dist Avg. 



Exhibit 3.2. Reading Proficiency by Cohort from 2004-05 to 2009-10 




Cohort 1 Cohort 2 - State Avg. Dist Avg. 
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Findings 

From the school year 2004-05 to the school year 2009-10, 

• Mathematics and reading achievement improved for Cohort 1 

• Mathematics and reading achievement were stable or dropped for Cohort 2 

• Neither Cohort 1 nor Cohort 2 showed substantial gains in student achievement during 
the CLA implementation period, although Cohort 2 shows a slight rise in mathematics 
achievement in this period 

• State test scores, especially in mathematics, rose during this five-year period 

These findings are consistent with many studies of educational interventions, especially those 
that are designed to influence student learning indirectly, by improving pedagogical and 
organization conditions for learning. Research suggests that it is remarkably difficult to move 
student test scores, and may take three to five years to have an impact (Aladjem et al., 2006; 
Borman, Hewes, Overman, & Brown, 2003; Comprehensive School Reform Quality Center, 
2006; Desimone, 2002). 

However, the CLA approach targets areas which research suggests have the greatest likelihood 
of eventually improving learning. For example, research on the social organization of schools 
found that professional community — shared responsibility, collective decision making, common 
values — when focused on student learning relates to instruction and achievement (Newmann & 
Associates, 1996). Researchers also found relationships between organizational conditions — 
such as teacher collaboration, teacher teams, principal support for teachers, flexible scheduling — 
and student achievement (Bryk & Driscoll, 1988; Lee & Smith 1995, 1996). One “bottom line” 
emerges from this research on effective schools and organizations: instruction matters most, and 
other changes (e.g., leadership, resources) also relate to student achievement when they facilitate 
changes in instruction (Gamoran et ah, 2000). 

Further research might follow two paths: (1) examine student outcomes after four, five, or more 
years of implementation and (2) examine effects of CLA on instructional strategies and other 
conditions for learning proven to relate to student achievement. Neither body of data was 
available for this exploratory study. 

It is also important to note that Cohort 2 began the study with very high levels of achievement in 
both subjects, and Cohort 1 began with very high levels of achievement in reading. It is possible 
that there is a ceiling effect; test scores were so high that it would be difficult to achieve higher 
regardless of the intervention. This is not atypical for studies using TAKS data. Future research 
might look at CLA achievement using other tests, to determine the degree to which a ceiling 
effect limits apparent impact. 



Discussion 

This exploratory study suggests that implementation of the CLA is generally strong, and that 
there may be a causal relationship between implementation and student outcomes. It is too early 
in the implementation to conclude that CLA improves student outcomes, and this exploratory 
study was not designed to make causal inferences or to make a definitive statement on impact. 
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Based on these preliminary findings, we suggest that a prospective study might provide more 
conclusive findings. This study should include pre- and post- intervention measures of 
implementation, should look at outcomes of the intervention after three years of implementation, 
and should use qualitative data to further explore the processes through which the intervention 
may affect the outcomes. Future research might also build on these preliminary findings by using 
more rigorous causal designs. For example, using carefully matched comparison groups — or 
even randomly assigning schools to CLA or a control condition — could provide a stronger case 
for the impact of CLA. 

Finally, in addition to looking at student impact more rigorously, further research on CLA might 
look more closely at the less concrete, but still critically important, impacts on school and district 
functioning and culture. According to district staff, and verified by U.S. Department of 
Education statistics, the district has undergone greater-than-average growth in its school 
population (see Appendix 4). Not only was Mansfield’s student population growth faster than the 
state average, but the district growth in students at risk (disadvantaged, limited English 
proficient, and/or minority students) was somewhat greater than the state average. The district 
feels that CLA was instrumental in building community and effective professional relationships 
under particularly trying circumstances. Additional research could explore the role of CLA in 
building community and the ways in which that may contribute to more effective district and 
school functioning. 
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Appendix 1: Failure Is Not an Option® Critical Success Factors 
Six Principles That Guide Student Achievement in High-Performing Schools 
Principle 1: Common Mission, Vision, Values, and Goals 

What does this mean? 



There is a eonsisteney in daily activities, policies, processes, and priorities, reflecting the 
uniqueness of the school in support of their autonomy. Schools are also accountable for results 
and contribute towards a common and coherent focus as a district- wide system. 

What does this look like? 



■ A representative stakeholder group is involved in the process of creating a mission statement. 

■ Mission, Vision, Values, and Goals are created through a process of consensus. 

■ The Mission, Vision, and Values guide decisions at all levels within the system. 

■ Actions and outcomes align with and support goals, values, and mission. 

■ There is an established communication process among and between schools to support and 
sustain system-wide efforts to accelerate student achievement. 



Principle 2: Ensuring Achievement for ALL Students: Systems for Prevention and 
Intervention 

What does this mean? 



Through a system-wide learning community, the process of designing and sustaining a 
continuum of support and instructional interventions for all students is developed and 
implemented. This includes School Improvement Plans (SIP) that identifies targeted priorities for 
instructional interventions and professional development plans that support implementation and 
monitor results. 

What does this look like? 



■ There are structures and processes to support capacity to build and sustain Professional 
Learning Communities (PLCs), focused on instructional improvements. 

■ Professional development supports building teacher capacity for instructional intervention 
strategies and expanding teachers’ repertoires. 

■ Communication of student data between schools and with parents/guardians supports the 
development of learner-focused, instructional improvement plans for all students. 

■ Schools, as a community, share the responsibility for closing the achievement gaps and 
ensuring success for all students. 
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Principle 3: Collaborative Teaming Focused on Teaching and Learning 

What does this mean? 



Faculty and staff focus their collegial conversations and ongoing professional learning on 
teaching practices to ensure academic success for all students. PLCs working collaboratively to 
improve teaching practices, is the norm. Professional structures such as school improvement 
teams, faculty/department meetings, networks and committees - provide the opportunities for 
collaboration. Processes provide the guidance and support to ensure its effectiveness. 

What does this look like? 



■ Teachers regularly open their classrooms to other teachers as opportunities to learn and 
improve their practice. 

■ SLTs display the characteristics of high performing teams and knowingly utilize structures and 
processes to support their team work. 

■ Effective team strategies used for continual team learning and improvement are monitored 
over time to assess progress and inform next steps. 

■ Collaborative problem-solving is embraced and seen as a vehicle to improve outcomes for 
every teacher and learner. 

Principle 4: Using Data to Guide Decision Making and Continuous Improvement 

What does this mean? 



Decisions to identify instructional priorities are based on analyzing multiple sources of data. 
School Improvement Plans are informed, and progress monitored for targeted instructional 
interventions. This process includes professional collaborations to continually assess results and 
make adaptations with instructional strategies, as indicated. 

What does this look like? 



■ Baseline data are used to identify SMART Goals as part of the instructional improvement 
planning process. 

■ Strategies are implemented and data collected using established procedures and protocols to 
monitor and evaluate progress. 

■ Decisions to revise, refine, or continue with instructional interventions are based on data 
analysis and the process for continuous improvement. 

■ Examining student work and observing teaching practices provides data, to continually inform 
instructional planning and professional development, is the norm. 
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Principle 5: Gaining Active Engagement from Family and Community 



What does this mean? 

School staff understands the importance of building positive relationships with their students’ 
families. Staff gains a common understanding of conditions that affect students’ learning and 
strategies for reaching out to the family and community to engage them in supporting their 
students’ learning. There is a shared understanding of the importance of building a partnership 
focused on the education of their child in addition to the traditional parent volunteer role. 

What does this look like? 



■ Communication practices between home and school are developed and utilized. 

■ Parents/guardians are represented in the School Improvement Plan process and are kept 
informed regarding baseline data and progress monitoring of results. 

■ A visible presence of school staff collaborating within the community is apparent. 

■ Family and community are involved in the school in a variety of ways that actively supports 
student learning. 



Principle 6: Building Sustainable Leadership Capacity 

What does this mean? 



Leadership extends beyond the formal school leader and is recognized as a means of ensuring 
commitment to sustain a long term vision. Lateral leadership is valued and strengthened as a 
means to implement and sustain changes that continually improve student achievement. 

What does this look like? 



■ There is a shared understanding of an established set of procedures, protocols, and resources to 
support continuous improvement over time. 

■ Job embedded professional development, to support courageous leadership development, at all 
levels of the school system, is part of the culture. 

■ Professional structures such as teams, networks, and committees, are audited to identify 
current collaborations including recommendations to review roles, membership, 
responsibilities, and impact. 

■ A plan for leadership succession is in place to assure sustainability of the established, 
collaborative culture. 
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Appendix 2: Sample Description 



The sample consists of 34 of the 40 schools in the Mansfield Independent School District, in 
Texas. There are a total of 38,947 students in the data set (duplicate records have been deleted). 
We used data from grades 3-12; grades pre-kindergarten through 2 are not tested. 



Cohort 2 


Cohort 1 


Elementary Schools 
(Grades preK/K-4) 


Intermediate 
Schools (Grades 5- 
6) 


Middle Schools 
(Grades 7-8) 


High Schools 
(Grades 9-12) 


Alice Ponder 


Cross Timbers 


Brooks Wester 


Mansfield 


Anna May Dalton 


Della Icenhower 


Danny Jones 


Mansfield Eegacy 


Carol holt 


Donna Shepard 


James Coble 


Mansfield 

Timberview 


Charlotte Anderson 


Mary Eillard 


Rogene Worley 


Mansfield Summit 


Cora Spencer 


Mary Orr 


T.A. Howard 




D.P. Morris 








Elizabeth Smith 








Erma Nash 








Glenn Harmon 








Imogene Gideon 








J.E. Boren 








Janet Brockett 








Kenneth Davis 








Eouise Cabaniss 








Martha Reid 








Mary Jo Sheppard 








Roberta Tipps 








Tarver Rendon 








Thelma Jones 








Willie E. Brown 
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Appendix 3: 


Failure is Not an Option Readiness Survey Levels 


Table A3.1 presents the school average Knowledge and Skill Readiness Levels for all Mansfield 
HOPE Schools, organized by cohort, and Table A3. 2 presents the Commitment and Action 
Readiness Levels. Both scales have a range of 0 to 48. Following the score is a label interpreting 


the meaning of the score. 


as assigned by Hope staff. 


Table A3.1 Knowledge and Skill Scores (Spring 2010) 




Score 


Ability Level 


Elementary Schools (Implementation started in 2008-09) 


Alice Ponder 


39.4 


Infused into the Work Culture 


Anna May Dalton 


40.6 


Infused into the Work Culture 


Carol Holt 


40.7 


Infused into the Work Culture 


Charlotte Anderson 


37.8 


Infused into the Work Culture 


Cora Spencer 


40.9 


Infused into the Work Culture 


D.P. Morris 


38.7 


Infused into the Work Culture 


Elizabeth Smith 


43.9 


Infused into the Work Culture 


Erma Nash 


35.7 


Partial Work Environment Integration 


Glenn Harmon 


38.6 


Infused into the Work Culture 


Imogene Gideon 


43.3 


Infused into the Work Culture 


J.L. Boren 


41.3 


Infused into the Work Culture 


Janet Brockett 


39.3 


Infused into the Work Culture 


Kenneth Davis 


39.8 


Infused into the Work Culture 


Louise Cabaniss 


38.0 


Infused into the Work Culture 


Martha Reid 


40.3 


Infused into the Work Culture 


Mary Jo Sheppard 


35.7 


Partial Work Environment Integration 


Roberta Tipps 


44.7 


Infused into the Work Culture 


Tarver Rendon 


39.9 


Infused into the Work Culture 


Thelma Jones 


38.1 


Infused into the Work Culture 


Willie E. Brown 


37.6 


Infused into the Work Culture 



Intermediate Schools (Implementation started in 2007-08) 



Cross Timbers 


37.1 


Infused into the Work Culture 


Della Icenhower 


38.1 


Infused into the Work Culture 


Donna Shepard 


43.0 


Infused into the Work Culture 


Mary Lillard 


35.3 


Partial Work Environment Integration 


Mary Orr 


38.0 


Infused into the Work Culture 



Middle Schools (Implementation started in 2007-08) 



Brooks Wester 


42.9 


Infused into the Work Culture 


Danny Jones 


35.4 


Partial Work Environment Integration 


James Coble 


39.3 


Infused into the Work Culture 
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Rogene Worley 


39.4 


Infused into the Work Culture 


T.A. Howard 


36.4 


Infused into the Work Culture 


High Schools (Implementation started in 


2007-08) 


Mansfield 


30.9 


Partial Work Environment Integration 


Mansfield Legacy 


35.1 


Partial Work Environment Integration 


Mansfield Summit 


36.1 


Infused into the Work Culture 


Mansfield Timberview 


33.2 


Partial Work Environment Integration 


All: 


38.7 


Infused into the Work Culture 



Table A3.2 Commitment and Action Skill Scores (Spring 2010) 




Score 


Willingness Level 


Elementary Schools (Implementation started in 2008-09) 


Alice Ponder 


40.3 


Check-points to maintain/improve 


Anna May Dalton 


41.3 


Check-points to maintain/improve 


Carol holt 


40.6 


Check-points to maintain/improve 


Charlotte Anderson 


38.4 


Check-points to maintain/improve 


Cora Spencer 


41.1 


Check-points to maintain/improve 


D.P. Morris 


40.0 


Check-points to maintain/improve 


Elizabeth Smith 


44.4 


Check-points to maintain/improve 


Erma Nash 


35.1 


Support/follow-up to sustain progress 


Glenn Harmon 


39.8 


Check-points to maintain/improve 


Imogene Gideon 


43.1 


Check-points to maintain/improve 


J.E. Boren 


41.6 


Check-points to maintain/improve 


Janet Brockett 


40.6 


Check-points to maintain/improve 


Kenneth Davis 


41.8 


Check-points to maintain/improve 


Eouise Cabaniss 


39.0 


Check-points to maintain/improve 


Martha Reid 


40.6 


Check-points to maintain/improve 


Mary Jo Sheppard 


36.7 


Check-points to maintain/improve 


Roberta Tipps 


46.0 


Check-points to maintain/improve 


Tarver Rendon 


41.3 


Check-points to maintain/improve 


Thelma Jones 


39.7 


Check-points to maintain/improve 


Willie E. Brown 


38.8 


Check-points to maintain/improve 



Intermediate Schools (Implementation started in 2007-08) 



Cross Timbers 


38.2 


Check-points to maintain/improve 


Della Icenhower 


39.4 


Check-points to maintain/improve 


Donna Shepard 


42.9 


Check-points to maintain/improve 


Mary Eillard 


36.8 


Check-points to maintain/improve 
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Mary Orr 



38.9 



Check-points to maintain/improve 



Middle Schools (Implementation started in 2007-08) 


Brooks Wester 


42.6 


Check-points to maintain/improve 


Danny Jones 


34.7 


Support/follow-up to sustain progress 


James Coble 


39.3 


Check-points to maintain/improve 


Rogene Worley 


41.9 


Check-points to maintain/improve 


T.A. Howard 


36.3 


Check-points to maintain/improve 


High Schools (Implementation started in 


2007-08) 


Mansfield 


31.3 


Support/follow-up to sustain progress 


Mansfield Legacy 


36.7 


Check-points to maintain/improve 


Mansfield Summit 


35.9 


Support/follow-up to sustain progress 


Mansfield Timberview 


33.5 


Support/follow-up to sustain progress 



All: 39.4 Check-points to maintain/improve 
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Appendix 4: Mansfield ISD Growth 
Table A4.1: Student demographics in Mansfield ISD, from 2006-07 to 2009-10 





2006-07 


2007-08 


2008-09 


2009-10 


Total 


100.0% 


100.0% 


100.0% 


100.0% 


ECE 


0.2% 


0.2% 


0.3% 


0.3% 


Pre-K 


1.9% 


2.2% 


2.3% 


2.4% 


Kindergarten 


7.7% 


7.1% 


7.4% 


7.5% 


1st 


7.9% 


7.9% 


7.6% 


7.7% 


2nd 


7.6% 


7.8% 


7.8% 


7.4% 


3rd 


7.7% 


7.6% 


7.9% 


7.8% 


4th 


7.6% 


7.7% 


7.6% 


7.7% 


5th 


7.7% 


7.6% 


7.6% 


7.6% 


6th 


7.8% 


7.8% 


7.6% 


7.5% 


7th 


7.7% 


7.8% 


7.9% 


7.7% 


8th 


7.6% 


7.7% 


7.8% 


7.6% 


9th 


8.8% 


9.1% 


8.8% 


8.8% 


10th 


7.6% 


7.3% 


7.3% 


7.3% 


11th 


6.7% 


6.8% 


6.7% 


6.9% 


12th 


5.7% 


5.4% 


5.4% 


5.7% 


African American 


25.1% 


26.3% 


27.3% 


27.5% 


Hispanic 


19.3% 


20.3% 


20.6% 


21.0% 


White 


49.7% 


47.1% 


45.5% 


44.4% 


Native American 


0.5% 


0.5% 


0.5% 


0.7% 


Asian/Pac. Isl. 


5.4% 


5.8% 


6.1% 


6.5% 


Econ. Disadv. 


29.1% 


30.3% 


32.7% 


36.2% 


EEP 


9.0% 


9.8% 


10.0% 


10.4% 


Disc. Placements 


1.7% 


1.8% 


1.7% 


16.3% 


At Risk 


33.2% 


34.7% 


33.7% 


33.6% 
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Table A4.2: Student 


demographics 


in Texas, from 2006-07 to 2009-10 






2006-07 


2007-08 


2008-09 


2009-10 


Total 


100.0% 


100.0% 


100.0% 


100.0% 


ECE 


0.3% 


0.3% 


0.3% 


0.3% 


Pre-K 


4.1% 


4.1% 


4.2% 


4.4% 


Kindergarten 


7.7% 


7.7% 


7.7% 


7.6% 


1st 


8.1% 


8.1% 


8.0% 


7.9% 


2nd 


7.7% 


7.9% 


7.9% 


7.8% 


3rd 


7.6% 


7.6% 


7.8% 


7.7% 


4th 


7.4% 


7.4% 


7.5% 


7.6% 


5th 


7.4% 


7.4% 


7.5% 


7.5% 


6th 


7.3% 


7.2% 


7.3% 


7.3% 


7th 


7.2% 


7.3% 


7.3% 


7.3% 


8th 


7.4% 


7.1% 


7.3% 


7.2% 


9th 


8.7% 


8.5% 


8.2% 


8.1% 


10th 


7.1% 


7.1% 


7.0% 


6.9% 


11th 


6.3% 


6.3% 


6.4% 


6.4% 


12th 


5.7% 


5.8% 


5.8% 


5.9% 


African American 


14.4% 


14.3% 


14.2% 


14.0% 


Hispanic 


46.3% 


47.2% 


47.9% 


48.6% 


White 


35.7% 


34.8% 


34.0% 


33.3% 


Native American 


0.3% 


0.3% 


0.4% 


0.4% 


Asian/Pac. Isl. 


3.3% 


3.4% 


3.6% 


3.7% 


Econ. Disadv. 


55.5% 


55.3% 


56.7% 


59.0% 


EEP 


16.0% 


16.7% 


16.9% 


16.9% 


Disc. Placements 


2.4% 


2.4% 


2.2% 


13.6% 


At Risk 


48.3% 


48.4% 


48.3% 


47.2% 
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Table A4.3: Student growth rates in Mansfield ISD, from 2006-7 to 2009-10 

2006-07-> 2007-08 2007-08-> 2008-09 2008-09-> 2009-10 



Total 


6.0% 


3.7% 


3.0% 


ECE 


35.8% 


30.6% 


1.1% 


Pre-K 


25.5% 


6.0% 


7.5% 


Kindergarten 


-1.5% 


7.8% 


4.6% 


1st 


6.6% 


-0.6% 


4.1% 


2nd 


7.9% 


3.8% 


-2.5% 


3rd 


4.6% 


7.5% 


2.0% 


4th 


7.2% 


3.3% 


4.1% 


5th 


5.1% 


3.8% 


3.4% 


6th 


6.0% 


0.3% 


2.1% 


7th 


7.6% 


4.5% 


0.6% 


8th 


7.6% 


4.9% 


1.2% 


9th 


9.7% 


1.1% 


3.1% 


10th 


2.3% 


3.7% 


3.2% 


11th 


8.1% 


3.1% 


4.8% 


12th 


-0.6% 


3.8% 


9.1% 


African American 


11.0% 


7.6% 


3.6% 


Hispanic 


11.6% 


5.2% 


4.7% 


White 


0.5% 


0.2% 


0.4% 


Native American 


4.9% 


2.6% 


39.4% 


Asian/Pac. Isl. 


13.7% 


9.2% 


10.4% 


Econ. Disadv. 


10.5% 


11.6% 


14.2% 


EEP 


15.3% 


6.1% 


6.5% 


Disc. Placements 


8.4% 


-0.9% 


881.9% 


At Risk 


10.8% 


0.8% 


2.5% 



Page 24 



Table A4.4: Student growth rates in Texas, from 2006-7 to 2009-10 

2006-7-> 2007-8 2007-8-> 2008-9 2008-9-> 2009-10 



Total 


1.6% 


1.6% 


2.0% 


ECE 


-1.7% 


-0.8% 


7.6% 


Pre-K 


3.2% 


3.4% 


7.0% 


Kindergarten 


1.1% 


1.6% 


2.0% 


1st 


0.9% 


0.9% 


1.1% 


2nd 


3.7% 


1.3% 


0.8% 


3rd 


2.6% 


3.6% 


1.4% 


4th 


1.7% 


2.6% 


3.5% 


5th 


2.6% 


1.9% 


2.5% 


6th 


0.6% 


2.7% 


1.9% 


7th 


2.7% 


0.8% 


2.4% 


8th 


-1.7% 


3.3% 


0.8% 


9th 


0.2% 


-2.3% 


1.0% 


10th 


1.7% 


0.0% 


0.8% 


11th 


1.5% 


3.0% 


2.2% 


12th 


3.3% 


2.1% 


3.6% 


African American 


0.4% 


0.9% 


1.1% 


Hispanic 


3.5% 


3.2% 


3.5% 


White 


-0.8% 


-0.7% 


-0.1% 


Native American 


2.9% 


2.6% 


13.5% 


Asian/Pac. Isl. 


6.0% 


6.6% 


6.0% 


Econ. Disadv. 


1.2% 


4.3% 


6.2% 


EEP 


5.9% 


3.2% 


2.0% 


Disc. Placements 


0.6% 


-5.3% 


533.4% 


At Risk 


1.9% 


1.6% 


-0.4% 
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